Skip to content

Fix flaky consensus smoke test#3016

Merged
CHr15F0x merged 19 commits intomainfrom
chris/flaky-smoke-test
Sep 23, 2025
Merged

Fix flaky consensus smoke test#3016
CHr15F0x merged 19 commits intomainfrom
chris/flaky-smoke-test

Conversation

@CHr15F0x
Copy link
Member

@CHr15F0x CHr15F0x commented Sep 17, 2025

Fixes: consensus smoke test flakiness

  • Pathfinder simulates proposals by using a dedicated separate DB for proposal creation only, that DB is only created if --consensus.enable=true.
  • When a proposal is decided upon, the finalized block goes into 2 DBs: normal storage, and the fake proposals DB (because newer proposals do depend on the older proposals if H>10, due to system contract 0x1 and we always want to generate valid proposals).
  • All processes are now terminated even if the test fails at any step (even setup).
  • All steps of the test setup process are also wrapped with timeouts as they can also fail.
  • Github CI is pretty unfriendly if the test fails, ie. it's not easy to access stdout and stderr dumps for all processes (they're in /tmp/...), so if the env variable PATHFINDER_CONSENSUS_TEST_DUMP_CHILD_LOGS_ON_FAIL is set all of those log files are dumped to the parent process descriptors should the test fail. We cannot use Stdio::piped() because the logs from all child processes would get mixed up and thus become unintelligible.

@CHr15F0x CHr15F0x force-pushed the chris/flaky-smoke-test branch 4 times, most recently from cb3b51f to fb2e808 Compare September 18, 2025 09:39
@CHr15F0x CHr15F0x marked this pull request as ready for review September 18, 2025 11:44
@CHr15F0x CHr15F0x requested a review from a team as a code owner September 18, 2025 11:44
@CHr15F0x CHr15F0x changed the title Fix flaky consensus smoke test, always select the proposer provided via config Fix flaky consensus smoke test Sep 18, 2025
@CHr15F0x CHr15F0x marked this pull request as draft September 18, 2025 19:29
@CHr15F0x CHr15F0x force-pushed the chris/flaky-smoke-test branch 3 times, most recently from bfc8143 to a84cc0d Compare September 18, 2025 19:49
@CHr15F0x CHr15F0x marked this pull request as ready for review September 19, 2025 09:15
@CHr15F0x CHr15F0x force-pushed the chris/flaky-smoke-test branch from fa5cfb3 to c421232 Compare September 19, 2025 09:20
@vbar vbar self-requested a review September 19, 2025 12:39
@CHr15F0x
Copy link
Member Author

Still fixing an issue with the fake proposals DB getting into a locked state...

@CHr15F0x CHr15F0x force-pushed the chris/flaky-smoke-test branch from 1a8cdd4 to 56af494 Compare September 19, 2025 15:28
@CHr15F0x CHr15F0x marked this pull request as draft September 22, 2025 17:09
@CHr15F0x
Copy link
Member Author

CHr15F0x commented Sep 22, 2025

Now that I'm more sure I fixed fake proposal creation I have discovered a bug which I'm fixing rn: #3019
Regardless the test is not flaky anymore and can be merged. Unfortunately the test may fail in CI until the bug is fixed.

@CHr15F0x CHr15F0x marked this pull request as ready for review September 22, 2025 20:39
@CHr15F0x CHr15F0x force-pushed the chris/flaky-smoke-test branch from da631bb to 0893a1a Compare September 23, 2025 07:51
@CHr15F0x CHr15F0x merged commit 6245d72 into main Sep 23, 2025
8 checks passed
@CHr15F0x CHr15F0x deleted the chris/flaky-smoke-test branch September 23, 2025 08:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants